{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Logistic Regression: Training stability\n",
    "\n",
    "## (a)\n",
    "The model training does not seem to converge when trained on dataset B.\n",
    "\n",
    "## (b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import src.util as util\n",
    "import numpy as np\n",
    "#import src.p01_lr as p1\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xa, Ya = util.load_csv('data/ds1_a.csv', add_intercept=True)\n",
    "Xb, Yb = util.load_csv('data/ds1_b.csv', add_intercept=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot dataset A:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dbYxcV3kH8P+za7loW7ArbCQUZ2eDmlSEqDLsKpAvLZQXmXywv1CaaBGgpliBhky8UqVQS+wqyFILqp2NGrXdtoiXXRICH8CiQanaBoGiJM1a2QQSlMiYtbMKahaILVVRCN55+uHMzd6ZvW8zc8+955z7/0kj77x49tyZ2WfOfc5zzhFVBRER+W+s7gYQEVE5GNCJiALBgE5EFAgGdCKiQDCgExEFYlddv3jfvn06NTVV168nIvLSmTNnfqmq+5Puqy2gT01NYXV1ta5fT0TkJRE5n3YfUy5ERIFgQCciCgQDOhFRIBjQiYgCwYBORBSI3IAuIl8WkZdE5Ccp94uI3CMiZ0XkaRF5V/nNJCKiPEV66F8BcCjj/g8DuLp7OQrgH0dvFhERDSo3oKvqDwH8OuMhRwB8TY3HAOwVkbeW1cCMhmVfJyJqmDJy6FcAeCF2faN72w4iclREVkVkdXNzc/jfuLAAHDu2HcRVzfWFheGfk4jIc2UEdEm4LbG7rKpLqjqjqjP79yfOXM2nCly8CCwubgf1Y8fM9YsX2VMnIgDAygowNQWMjZl/V1bqbpF9ZUz93wBwZez6AQAvlvC8yUSAU6fMz4uL5gIA7ba5XZK+X4ioSVZWgKNHgVdeMdfPnzfXAWB2tr522VZGD/00gI93q13eA+CSqv6ihOdNJwKcPNl728mTDOZEhJUV4BOf2A7mkVdeAY4fr6dNVSlStngfgEcB/KGIbIjILSJyq4jc2n3IgwDOATgL4F8AfMZaayPz88D0dO9t09PmdiJqrKhnvrWVfP+FC9W2p2q5KRdVvTnnfgXwV6W1KE+nA5w+DaytAQcPAmfOmGC+tmbun583STMiapzjx3f2zOMmJ6trSx38i3xjY8DhwyaYr60B4+Pbwf3wYazcN9a4gRAiMrJ64BMTwIkT1bWlDqI1VYXMzMzoSOuhdzommEe2trBy31jPQAhg3sSlpbAHQojImJoyA6D9xseBr341jDggImdUdSbpPv966IApTZyb671tbg7H/0ZLHwhpYukTka9OnDCduLiJiXCCeR7/Anq87rzdNj31dhtYXEw93Rp2ICQaYDl/3vzaqPSJQZ3ITbOz5oy81TJFb61Ws87Q/QvoIsDevb1156dOAe02JvdcSvwvww6EJA2wuFD6xLMGonSzs8D6uunrra83J5gDPgZ0wEzxj08i6gb1E/fuTTzdGnYgpOwefxl41kBVYcfBP34GdGDnJCKR0k+30nr2dZY+uXrWQG4bNDiz4+ApVa3lMj09ra5bXladmFA1H2lzmZgwt9dFpLc90UWkvjYlWV5WbbVMu1qtel+zphvmc9xqJX/OWq2qWk1pAKxqSlz1t4deARcHWFw8a+jH3p1bhjmrczHdSPn8rUNvqP5FhwD3au3TaoFbLTNIRdUaG0tehFTEDBwm4XvorvDq0BvMxbOGfuzduWWYs7q0eu7QZ1r6jgHdQ66XZfmQFmqSYYKzDx0H2okBnUrH3p1bhg3OrnccqjZMpVDlZZ9po6W2Lz5UudDwWOVCIRm0UshmhRwyqlw4KEpElGPQQWKbg8ocFKXG4mxHylPkMzLoQH9dhQEM6BQs1sNTnqKfkUEH+usqDGBAp2BxmQTKU/QzMuhAf12FAQzoFCzWw1Oeop+RQSuF6ir75KAoBYuzHSmPj58RDooGhgN9xbAenvKE9hlhQPcMB/qK42xHyhPaZ4QpF8/4eIpIROVhyiUgHOgjojQM6FXoPwsa4ayIC18RURoGdNsWFoBjx7aDuKq5vrAw1NOFNogzDA4KEyVjQLdJFbh4EVhc3A7qx46Z6xcvDtVTD20QZ1AcFCZKx0FR2+JBPNJuA6dO7dzomnJxUNgPKytmtuWFCyYdeOJEczodtmUNijKgV0HV5AcinQ6D+ZCG2U6NquXDNok+Y5VLnaIeelw8p04D4aCw+7iGTn0Y0G2Kp1vabdOFbLd7c+o0kGEGhTmIWi2W1tanUEAXkUMi8pyInBWROxPunxSRh0XkSRF5WkRuLL+pHhIB9u7tzZmfOmWu793LtMsQBh0U5iBq9XgW1avSDkXaVkbRBcA4gJ8BeBuA3QCeAnBt32OWAHy6+/O1ANbznrdRW9B1OtnXyZpWq3cbsOjSatXdsnDZ3H7NNzZeC2RsQVekh349gLOqek5VXwNwP4Aj/d8LAN7U/XkPgBdH+ZIJTn9PfNieeYkTlPqFmpbg6X/1ml5aG1f1eMKuAo+5AsALsesbAN7d95gFAP8hIp8F8LsAPpD0RCJyFMBRAJhs6vnXsBYWTO16lLqJ8vN79w49SSnSX5UQpSUA//8IJyeTyxz58bNrdtb/z04Zqu5QFOmhJ3Un+7uGNwP4iqoeAHAjgK+LyI7nVtUlVZ1R1Zn9+/cP3lrfDdvDtjBBKS7kqgTOrKU6VT2eUCSgbwC4Mnb9AHamVG4B8AAAqOqjAN4AYF8ZDQzGKEsAxAdTFxdNXiSqnClhglLIaQme/lcr1NTdsCrvUKQl16MLTFrmHICrsD0o+o6+x3wfwCe7P78dJuBL1vM2blC03TYjIu128vWizxMfXSlpcJUDh1QGDoYmW142f0si5t9RXw9kDIrmBnTz/3EjgOdhql2Od2+7C8Bh3a5seaQb7NcAfCjvORsV0FV7g3h0GTSYj/L/M/APkcrAjkE1Rg7oNi6NC+iqw/ewy+rhZyi7F0HNI5Ic0EXqblk+nz7/WQGdM0WrEuXM44rOFq1ggtLsrFncqtMx/zLHTIPydULRIJPPnB8jSIv0ti+N6qGXmUPPuk5UI19Td0VTRa4cH9hDr1lZPeyyJigRWVCkosjFHm7RKi8fynu5fG6VVHuDcP/1EDThGGkori6rW3SNfVeWbm728rn970BNX2AAwu9hl7zdHoXF1R5u0VpxH8YIwg7oDDDVUbuzWcl/rk5gKzr5zItZx2nJddsX64OiFZT6UR+LtfLkvxDq1F0ob0TGoGjYOfR4LzHiyn6eoeaaVbndHiVyNYfum+bm0KNqkjgXgnmoqaBRau3JaWVUp3BdHfvCDuguBphQc83x4+B2e5lcLN3LUuauT6NOYPPttatcWi7G9iXoHHreBKBQc83z873HER3n/HydrXKKK5NTBuFK7tvH107VwcW5bFwqmSlaR4Ap+jstrZxYO85mzZQXHF0YdOvnyhotrnyxDKLqLejCDuiq1QaYomcFofbQKVdWcHS1B+pKIHXli2UQNl67Zgf0quUFa5ZTOqOO3nDWH7grgbOfK180rr4+WWx8CTGgVy0vnTJIWibrOg2triCV9Xtd7oG6kApy5YtlEOyh+65oOiXvelMGGGv60qqzt5cWHH3sgVbNhS+WQTCH7rMyl8ltQlqmxi8tF3vDPvZAKR+rXHxWVpAKfeC05i8tV3vDvvVAqXoM6FUrK41Qcmmjc8Gixi+tJvSGnXu/qRQM6D4qM9h1Ou4GsBrr8UMOeM6+3zQyBnTflJmO6KaAWpMd91IMoaeVauRqSolGlxXQw17LxVdlbVmn+vq6Mc6tRa3KtV8scu79pkrsqrsBlGJhwQS1KHhHQX2QlSJjq01OLp7HeUzteIiV3Vbi7U66HrUt6UsLGOxLi163smJ2/7lwwSxetbW18zEu7a5Tl/jrNDlpNqgIZsXHtK677QtTLhXqdHQZN+sE/s9+TnXQKh9OnipFUs68/8IcehhjC2DKpcG6qY1Z3IclfAotrEOgaE1q+WtRx1I8hZcGDn2f1Yok7dcJAOPjXHs8ztV9TUuTFultX9hDr0Adtd4c6KyFjYlSIVYBuTihbFBgD72hyhpcHfR3urhLVODK3pG+zE0tXFL26+SctEhv+8IeeoXqWkKYPfTKlJ0bDrXskTl08l9Veep4zpyliJUqe7/OUMseQ9/XlGWLVB6WItZqdra8wDQ5adIsSbf7rszXyTUM6FSuMurnqXYnTpicebwiZGLC3E7uKpRyEZFDIvKciJwVkTtTHvNREXlWRJ4RkW+U20zaka5wOX3BUkTv1Z2aWFkBpqbM7961y/w7NeX/oKxtojmBQUTGATwP4IMANgA8AeBmVX029pirATwA4E9V9WUReYuqvpT1vDMzM7q6ujpq++sT74UmXS/TwoKp4456ulGueu9ecx9RQKIKm6S6+omJsHLewxCRM6o6k3RfkR769QDOquo5VX0NwP0AjvQ95lMA7lXVlwEgL5h7b2Ghd5AvCrA2guswk3WIPJY2SQoIbBKQBUUC+hUAXohd3+jeFncNgGtE5BEReUxEDiU9kYgcFZFVEVnd3NwcrsV1qzrAxmvHFxfNIh1RFcnJk0xnUHDyKml8r7SxqcigaFLE6I9auwBcDeC9AA4A+JGIXKeqF3v+k+oSgCXApFwGbq0L4pUbi4vmAvRWdtj4nXv29N528iQwN8e0CwUnrcImfj8lK9JD3wBwZez6AQAvJjzmu6r6W1X9OYDnYAJ8mKqeDdnpAKdP9942Pc20CwXpxAmTK0/CSptsRQL6EwCuFpGrRGQ3gJsA9EUXfAfA+wBARPbBpGDOldlQp0RpljhbE2dUTU98bQ04eHD79ug60y4UmHiFDWAWGAPCmwRkQ25AV9XLAG4D8BCAnwJ4QFWfEZG7RORw92EPAfiViDwL4GEAf62qv7LV6FpVPRsyPlnnzJne+w4fNjl1osDMzgLr6+bP6fJl8+/6OoN5nkITi1T1QQAP9t32+djPCmCuewlbHbMhFxbMF8dc38t76ZLdckkqLOhNE8gbnCk6jKpnQ0Zpl+is4NSp7bMEgDMxa9ZfNx2tTAgwqFO1GNCHVeVsyKatkVLlpK0SZG2awIBOVcqdKWqL9zNF6zBMoPMsOPo4K3ZsLH0zpk6n+vZQ2EadKUquGPSsoMoZrWXwdFZs8JsmeCZaB2ZsrHnrvzCgh8rH4Jg1K9bhcYKkumnWS9cj1J2WimLKJWTxIB5xPDgCMO2Ol2N2Om63F6xyccW+fcCvEgqmWy1T9hgCplyaysf9PauctFWiqG6602G9dF1WVpKDOdCc9V8Y0EPmW3DkFnY0gqxVGJsynsGAHiofg2NaeWa7HWZ5JpUqqxfelPEM1qGHytfadW5hR0NKW6XxzW9uTgqMAT1kvgZHbmFHQ0jbBzVeExA6plxCx+BIDVH3PqguYA+diIIxO9usAN6PPXQiclaTZ30Ogz10InISV7EcHHvo5Kb+skoXyyzJqqxVLCkZA3ocg4gbfFtUjKxIqytvyqzPYTCgR+oOIvwyMXxcVIys4CqWg2NAB+oPInV/mbjE0xUXqXxcxXIIqlrLZXp6Wp3S6ai226omnJpLu21ur+r3Rr+v/3rW/8267rNOp/e9COnYGm55WbXVUhUx/y4vl/PYpgCwqilxlQE9rj+IbG3tvN/W7x30y2R+vvcx0XPMz9tpY5nyvojq+nIl65aXVScmet/aiQkG6kFkBXSmXCJJKxNOT2/vIWYzDTLoMrd1p4hGkZde8nFRMSqMlSt2MaADO4PI1hZw8CCwtrYd1G0GzKQvk6zg5WueucgXEVdcDBorVyxL67rbvjiXculPYWxtqR48aP+0f9Qcum955qLplJDHBxqs1ep966NLq1V3y/yBjJQLt6CLi3qIkU4HGB/vvW6jhzjMTvfx3m3E9R56RP3bYo7K0T/7EzCVK01bRGsUWVvQsYeepuqBuUF6pKP06uvGAc/GY+XKaMBB0QHVMTA3yDK3vuaZOeBJ4P6rNnFxriQ+7Pbj4+YVPryuRB5jDj1Lf069/zoNh69rNfg6Bykrh86USxbu9mMHX1f7uJxEIzGgE4WmSL0/BalQQBeRQyLynIicFZE7Mx73ERFREUkuqSEi+zyceMadicqRm0MXkXEAzwP4IIANAE8AuFlVn+173BsB/DuA3QBuU9XMBLkXOXRqhlBzzZ7U+7M2fTCj5tCvB3BWVc+p6msA7gdwJOFxXwDwRQCvDt1SGkz/lzFPpQcXaq45Oo44R0tDub5LeYoE9CsAvBC7vtG97XUi8k4AV6rq97KeSESOisiqiKxubm4O3FhvVBFoQw1EVQo11+xZvT/XdylPkTr0pHO01z8RIjIG4BSAT+Y9kaouAVgCTMqlWBM9M8w0/kHFAxFgflf8DziUlIFt8Tr4xcXt19PhXHMhntX7T06aDaCTbqcBpU0hjS4AbgDwUOz65wB8LnZ9D4BfAljvXl4F8CKAmazndX7q/zCqnJLPKfTl8XGRsyI8WeCMa6QPBqNscAHTiz8H4CqYAc+nALwj4/E/yAvmGmpAV6020IYaiKrEL0YncH2X4rICem4OXVUvA7gNwEMAfgrgAVV9RkTuEpHDJZwkhGXQzSqG5dGgl7M8yzWHjOu7lKPQWi6q+iCAB/tu+3zKY987erM8lhZoywzq/YEonkMH/M7/VsmzXDNRHi7OVaaqAm0VgSipNjv63WmP8ZGPi5wRpWBAL1OVPT6bgSipUueGG8x9jz5qr3qnLlxbhgLBgF62Knt8NgJRUknkHXcAjz9urt9xB3D33SyTJHIQl8+lneKpo8jtt5t/77ln+zbf67WJPJQ19Z8BnZIlrQMCeLE2SGOEugYNZeJ66DSYpEqdO+4wlziW9tWHSz9QAgZ06pVUm3377SbVcs895mfWa2ezvZZPqGvQ0MjCHxTlaelgkip17r57e1D07rtZr52lirV8Ql2DhkYWdg69ij+uUDWlDr1MWfMQbATbpHEOvh/Ba2YOnael2fLSAkklkazXzlblTkFc+oGSpC3yYvtSyeJcXHgp2fx87+sQvU7z83W2Khy2F02rclVPcg5GWZzLa1UtlOUTnrnYVUXPOW1GcrudO6bBvTsDlxbpbV/YQ68RXxc7qu45D7jeOdcdDwMa2UNPKr9jqZ3BMxc7Rug5D/37sq734d6d4Qu3bJFLo6ZLSwswqA8vqviJ1vKJOLR6I/fuDF+4AR3g0qhJskrrAL4+w+gvjwV6y2MdeT25d2f4wk25RFhq16vqtEDoPBpkPnECmJjovW1iwtxOYQh7YhGl4wza8sSDeMTRWZsrKyZnfuGC6ZmfOMHt3nzD1RaJbOOsTapIM2eKElWFszbJEQzoRKNgeSw5JOwqFyLbWB5LDmEOnagMHGSmijCHTmRbgOWxXPfFP0y5ENEOKyvA0aPbSwWcP2+uAyxzdBl76ES0A9d98RMDetPlbXRBjcR1X/zEgN5k3DmeUqSt78J1X9zGgN5UHq1BQtXjui9+4qBoU3HneMoQDXxy3Re/FKpDF5FDABYBjAP4V1X927775wD8JYDLADYB/IWqJizUuY116I7gGiREXhmpDl1ExgHcC+DDAK4FcLOIXNv3sCcBzKjqHwH4NoAvjtZkqgTXICEKSpEc+vUAzqrqOVV9DcD9AI7EH6CqD6tqVOT0GIAD5TaTSsc1SIiCUySHfgWAF2LXNwC8O+PxtwD4/iiN8p4P08C5BglRcIr00JP+shO7byLyMQAzAL6Ucv9REVkVkdXNzc3irfSJT6WACwu9A6BRUHexrdRYXIKguCIBfQPAlbHrBwC82P8gEfkAgOMADqvqb5KeSFWXVHVGVWf2798/THvd5mMpYIBrkHiBE7oKiZYgOH/evETREgQM6ilUNfMCk5Y5B+AqALsBPAXgHX2PeSeAnwG4Ou/5osv09LQGqdNRbbdVzefPXNptczuRqur8fO9nIvrMzM/X2Sp7+j/7A/wttFq9f0rRpdUavVnLy+Z5RMy/y8ujP2cVAKxqSlzN7aGr6mUAtwF4CMBPATygqs+IyF0icrj7sC8B+D0A3xKRNRE5Xeq3jk/iuegI67opUvQsLpQe+4gpSFtLEATb80+L9LYv7KFTYyV9Rg4eVN3a6r3f9x57/Dijv4H+6zls9dBt9vxtQ0YPnQG9TCV8gKkhOp2d0STEz8yIHZzlZdXdu3v/++7do6dHRJIDushoz1uFrIDOtVzKlFYK2G6zFJC2RWmHuIMHTdplbGx7bkAIqboSUpD92acyslHBLj6WFultX4LsoUdGGAQix4363madxcUvRZ7Xh8/ZiD10W6mR5WXViYne55yY8GNgFOyhV4ylgGEqY45B0lncyZOmhx6XN1vXh/kOUZtGmI1sa1B0dhZYWgJaLfMWtFrmuu+LjzGgExWhJc4xiE/oUgXm5oC1teJBr8y22FRCCtJmamR2FlhfNy/5+rr/wRwAUy5EhdmqYBqmLt2VaqoiaZ8RUkM+p0ZsQUbKpdDyuTZw+VzyklpabliHWP/HVluKWlgwZwTxs41jx0zvu8TUz8oK12WPG2n5XCLqSqpOKWtlykHHXWy2pYgK0z5BpkZsSeu62740NuXiQ2UC7eTSHANX2uJK2qdhkJFy4RZ0VaroFJUscGm5YVfaEv3eaPtCIIzaeY8xoFclfooKmA9+vKSrSM6U6rWw0Ps+RQGtjvfNhbakpX0Y1GvDHHpV4iVbIc4IdEF/3tZGPtmlOQZ1tqWEGnMqHwN6lbgSoz0+TLQJCZe5cBJTLlXiKaodTGfVw4W0D/VgD70q/aeoW1u9p6idTt0t9BfTWfVxKQVFnFhUqajKZc8e4NIls4bH3Nz2dVa7jCZvos0wk3eIHMOJRa5YWDBB/NIl04Ocm+u97tI6HL7Jm2jDHDs1AAN61cbGetMD4+NMD4wqr+Ki0/FjMSuiETHlUpe61+EITd6krXgQj/R/iTIlQx7ISrkwoNehSHChweUF5KwvUc7iJU+En0OvYkJJWTghw56siousHHuFC00R2eR/HbpvPStX1uFokv4v0XidOmCuR+/B4uL27TxrIs/4HdB9nVDCCRnVyvsSjR7DhaYGwzEH5/idcvF5QgknZFQrvu0b0BvUo0qYOtcX9w3LQJ3kd0AHuD6K76oc/0j6TERneNPT5t9os+aDBzmukYZjDs7yP6DXvXMLDa/uXl78DG9tzdwWbdZ85kwzFpoa5gvV5zPj0KXtfGH7UsqORa7s3OKzunZQcum963R6d92Jb9YcsmE2p45Le93IKmTsWOR3D51LeI6mzh6yK728rDO8kD8/o6ZNeGbsprRIb/tS6p6i3KdzcK70kOvs5bnyGtRl2D1Bm/661QzB7ynKipHBxQeT66q9TuvlVfX7mz4nYNhSzaa/bi5Li/S2L6X20Gl4dfWQXerlNfUMb9geevz/Z10nKzBqDl1EDonIcyJyVkTuTLj/d0Tkm937HxeRqZK/d8iGOvOgLo1/NPEML54zH3YJiia+bo7LTbmIyDiAewF8EMAGgCdE5LSqPht72C0AXlbVPxCRmwD8HYA/t9FgKkn/H3TSdHjbf6CcMVsfpk2CVCSHfj2As6p6DgBE5H4ARwDEA/oRAAvdn78N4B9ERLqnB+QiV/6g2curD79Qg1MkoF8B4IXY9Q0A7057jKpeFpFLAN4M4JdlNJIs4R808Qs1KEVy6EnvcH/Pu8hjICJHRWRVRFY3NzeLtI9s4x80UTCKBPQNAFfGrh8A8GLaY0RkF4A9AH7d/0SquqSqM6o6s3///uFaTEREiYoE9CcAXC0iV4nIbgA3ATjd95jTAD7R/fkjAP6b+XMiomrl5tC7OfHbADwEYBzAl1X1GRG5C6Ye8jSAfwPwdRE5C9Mzv8lmo4mIaKdCM0VV9UEAD/bd9vnYz68C+LNym0ZERIPwe3EuIiJ6HQM6EVEgGNCJiAIhdRWjiMgmgPMjPs0+NGvyUtOOF2jeMfN4wzfqMbdUNbHuu7aAXgYRWVXVmbrbUZWmHS/QvGPm8YbP5jEz5UJEFAgGdCKiQPge0JfqbkDFmna8QPOOmccbPmvH7HUOnYiItvneQycioi4GdCKiQDgf0Ju4n2mBY54TkWdF5GkR+S8RadXRzrLkHW/scR8RERUR78vcihyziHy0+z4/IyLfqLqNZSrwmZ4UkYdF5Mnu5/rGOtpZFhH5soi8JCI/SblfROSe7uvxtIi8q5RfnLZ7tAsXmNUdfwbgbQB2A3gKwLV9j/kMgH/q/nwTgG/W3e4Kjvl9ACa6P3/a52Mucrzdx70RwA8BPAZgpu52V/AeXw3gSQC/373+lrrbbfl4lwB8uvvztQDW6273iMf8xwDeBeAnKfffCOD7MJsDvQfA42X8Xtd76K/vZ6qqrwGI9jONOwLgq92fvw3g/SJeb7uTe8yq+rCqvtK9+hjMpiO+KvIeA8AXAHwRwKtVNs6SIsf8KQD3qurLAKCqL1XcxjIVOV4F8Kbuz3uwcxMdr6jqD5GwyU/MEQBfU+MxAHtF5K2j/l7XA3rSfqZXpD1GVS8DiPYz9VWRY467Beab3le5xysi7wRwpap+r8qGWVTkPb4GwDUi8oiIPCYihyprXfmKHO8CgI+JyAbMUt2fraZptRn077yQQuuh16i0/Uw9Uvh4RORjAGYA/InVFtmVebwiMgbgFIBPVtWgChR5j3fBpF3eC3MG9iMRuU5VL1pumw1FjvdmAF9R1b8XkRtgNsy5TlU79ptXCytxy/Ueemn7mXqkyDFDRD4A4DiAw6r6m4raZkPe8b4RwHUAfiAi6zD5xtOeD4wW/Vx/V1V/q6o/B/AcTID3UZHjvQXAAwCgqo8CeAPMIlahKvR3PijXA3oT9zPNPeZuCuKfYYK5z7lVIOd4VfWSqu5T1SlVnYIZMzisqqv1NLcURT7X34EZ/IaI7INJwZyrtJXlKXK8FwC8HwBE5O0wAX2z0lZW6zSAj3erXd4D4JKq/mLkZ617NLjAaPGNAJ6HGSU/3r3tLpg/asC88d8CcBbA/wB4W91truCY/xPA/wJY615O191mm8fb99gfwPMql4LvsQA4CeBZAD8GcFPdbbZ8vNcCeASmAmYNwIfqbvOIx3sfgF8A+C1Mb/wWALcCuDX2/t7bfT1+XNZnmlP/iYgC4XrKhYiICmJAJyIKBAM6EVEgGNCJiALBgE5EFAgGdCKiQENiHNgAAAAMSURBVDCgExEF4v8BgXi6h79AhbsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "util.plot_points(Xa[:,1:3],Ya==1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot dataset B:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAev0lEQVR4nO3dbYxcV3kH8P/jBAMrQRbZRkVJdjZRHQmXDwm2CFGlFgQFY1XJl0ANG16qiBVUsOukqhRrK3aTyqKt1DgOpFALRUDsQmirwgqZWOJNqRBOs1GcQIKCnOAk2yCyFBKpMhTSefrhzMV3ZufeuTP3nnve/j9ptHtnr2funRk/c+5zznmOqCqIiCh8W1wfABERNYMBnYgoEgzoRESRYEAnIooEAzoRUSQudPXE27dv19nZWVdPT0QUpIceeujnqrpj2N+cBfTZ2Vmsra25enoioiCJyNNFf2PKhYgoEgzoRESRYEAnIooEAzoRUSQY0ImIIjEyoIvI3SLyvIj8sODvIiJ3isgZEXlURN7Y/GESEdEoVVronwewt+Tv7wKws3ebB/CZ+odFRETjGhnQVfV+AL8o2eU6AF9U4xSAaRF5XVMHSERE1TSRQ78YwLO57fXefZuIyLyIrInI2sbGRgNPTVUcPw7MzgJbtpifx4+7PiIisqGJgC5D7hu6aoaqHlXVPaq6Z8eOoTNXqWHHjwPz88DTTwOq5uf8PIM6UYyaCOjrAC7NbV8C4LkGHpcasLQEnDvXf9+5c+Z+IopLEwF9FcAHeqNd3gzgRVX9aQOPW0tKaYayc33mmeH/puh+IgrXyOJcIvIlAG8BsF1E1gEsA3gZAKjqZwGcALAPwBkA5wD8ua2DrSpLM2Qt0yzNAABzc+6Oy4ZR5zozY+4bNDPT3jESUTvE1SLRe/bsUVvVFmdnhwexTgc4e9bKUzoz6lwHAz4ATE0BR4/G9+VG5v1eWjJXYDMzwKFDfJ9jIyIPqeqeYX+LcqZoaZrB0ReYLaNSKnNzJnh3OoCI+clgHid2gFOUAb0onTBzqQI33QSsrLR6PDYVnmvu/rk501rvds1PBvM42egAT6kvKgZRBvRDh4Cpqf6W+NSU4tDr7wGOHAFeeCGalro51/77pqbM/ZSWpjvA2eIPT5QB3aQZBJ0ZhUDRwVkcPTeHuZMfBBYXgcOHTf5hQj61WphSoUyVq7VxcMhrgFTVyW337t3aim5X1TQwzK3brfxPjx1T7XRURczPY8fMbWqq/yGnpsz9ZM+w94L6Nf3ZFOl/rOwm0uxx03gArGlBXI07oHe7qouL/Z/GxcVKQb3oP8e2bcM/5J2O/dNJFb9Eq2vyi6/Tae+zzi/s6tIM6PlgngXxwe0SRR/mohtbLfa0GVjoPBtfpLzqra8soI+cWBQsEWB6uj9nfviw+dv09Mgc+rgdSZyoYw9nu7qR9cM0Na69aBLcK19ZnKtnX9B4opxY1Ee1P3gPbhcomrCzbRvwq1+lM1HHh4kqKU0Ui1nR+1hExAy1pX7JTSzqMxi8K45uKRoOeORIOqNKfBm2xqGZceBVbwuKcjG2b62Ncqkh9Y4an3LXqb8XMSj6PG3bxhz6OFCSQ48/5UIT27Jl+PwrXgrTJMrqCgHuU3uhKEu5xNspSrWxUiM1aVQnKwN4ffHn0GlizF1T01hXyC4GdCrEsgLx8ql8BTWHKZdhJhzqGKO5OQbw2KS0AExq2EIftLJiSuxmvYEaX8ldgC20lLHoVrwY0PNUTWndI0fOB/Wbboqu5K4v48vJDc68jRcDel5WHmBx0QTxLVvMzwZK7vqELbS0NV1mtw5eKTaLAX1QvuZLJqJgDrCFljpfRi/xSrF5DOiDsjRLXj6nHgGfWmjUPl9GL6VypdjqVUjRFFLbNy+n/tcsuRsKlislH7hYQKPtEhI2/q+hZOo/W+h5RSV3FxcrldwNhS8tNEpb21eKLlI8bV+FsJbLMByHTmRdWW0XG40LF2WYbdRDSrt87iQmLLlLRNW1faXoYjBA21chDOhE5EybtV1cDAZoe0QRAzoRJcHFcM22r0JYy4WIktD0GqnjPG9bAw4Y0IkoGbEXm2PKhYgaxen87lQK6CKyV0SeEJEzInLLkL/PiMh3RORhEXlURPY1f6hE5DtO53drZEAXkQsA3AXgXQB2AXiviOwa2O2vAXxFVa8CsB/APzZ9oFYMDhCNaHp/m9gio0wq0/l9VaWF/iYAZ1T1KVX9DYAvA7huYB8F8Ore7xcBeK65Q7QkkbrntrFFRnks/OZWlYB+MYBnc9vrvfvyVgDcICLrAE4A+PiwBxKReRFZE5G1jY2NCQ63IYnUPW8DW2SUx8JvblUJ6MOmSQ5GvPcC+LyqXgJgH4B7RGTTY6vqUVXdo6p7duzYMf7RNiWRuudtYIuM8nwpzZuqKgF9HcClue1LsDmlciOArwCAqn4fwCsAbG/iAK1JoO55G9gic8+nPgwWfnOrSkB/EMBOEblMRLbCdHquDuzzDIC3AYCIvB4moDvMqVSQQN3zNrBF5paPfRhtTuenfiMDuqq+BOBjAE4C+BHMaJbHROQ2Ebm2t9tfAviwiDwC4EsAPqSuyjhWkc+ZLy6aT16WfmFQHwtbZG6xD4Py4iufW7X07cqK6QDN0ixZkJ+e5kgXCoaN8qzkt7LyuXFN/R8nSK+s9Af7LKfOHDoFZGZmeI1v9mGkKZ6p/5MMRWTdcwoc+zAoL54Wen7UypEj5gbYGYrIFY3IE64qCJKf4syhb8ldeHS7zQZb5t6JyKF0lqCzPRSRM0yJyGPxBPQ2hiJyhilRK3yaLBWSeAK6iEl75INrFnynp5sLtpxhmiwGmXb4OFkqFHHm0G12WOavBDJsoUcvCzL5STxTU5xEZcPs7PChmJ2OmXmaunRy6IDdoYicYZoszshsDwu+TS6+gG5TW2kd8g6DTHtiK/jWZqouvpRLGzgOPTlMA7SnLL0F9I+537cPOHHC3zH4NlJ1aaVc2sAZpsnhjMz2FBV8AzZ3ln7mM353nradqmNAJ6qAVSXty6cmlpbMl2W+BO+w4DjIt36NtlN1TLkQkXNVUhNFlSUH+VRp0kaqjikXIvJaldRE1U5RnzpP207VMaATkXNVUhPDguOgqSnTUerLBLC2U3XxVFskomBVqes+rLLk4CiXffuAL3zhfGs/6yjN//u2zc2199zMoYeMwycpEk0N70theClz6DFaWemfnZrNYmUJ39pYs6V9TaUmUp8AxoAeIpbxtYaFodyZmzOt6PxQxXHFNst0XEy5hIpFwqxI4ZI9ZikUUStLuTCgh6zu6kzMwW9SNNbZp7HNVO748biX5GMOPUZ1V2diDn6o1C/ZY9BE6iZUDOjA5iDoew66bhlfxzl4nzsdWbMlXT5/LitTVSe33bt3qxeWl1UXF1W7XbPd7Zrt5WWXRzVa3ePO9jfh29zyj2fJsWOqU1P9Tzs1Ze73xbFjqp2Oqoj56dOxkR0hfC4zANa0IK6mnUMfbOkePrx52+ecct0ceN0c/ATY6Ug+CulzyRx6kdAXfa5TxrduDr7oMcu2wXHC5KdYPpdpB3QgzUWfbSylV7GTlZ2O/aLI20Ygls8lA7qNlqrvml5Kb4xOVnY6npfqJCYfv8Rsfi5bPd+i5LrtmxedovmOwaxDcHA7ZoPnV+d8x+hkZaej0en0v1zZrdNxfWT2+Nz5aONzaeN8UbdTVET2AjgC4AIAn1PVvx2yz3sArABQAI+o6vvKHtOLTlHApAReeOF8SzVrXU5PJz8me2wOOll9VHViS4qTmELqfGxC2wtcjGxJwwTxJwFcDmArgEcA7BrYZyeAhwG8prf92lGP60ULPdNkSzVVjoZB+macFlmKLXSR4ecs4vrI7LBxvihpoVfJob8JwBlVfUpVfwPgywCuG9jnwwDuUtVf9r4knh/rK8c1Lvpcj41O1kCNsyhwiv0JsXQ+VtX2+VYJ6BcDeDa3vd67L+8KAFeIyPdE5FQvRbOJiMyLyJqIrG1sbEx2xOSfpjtZAzbO8LcUF54e9iUmYtISvnSQNqn1L+2ipnt2A/BumLx5tv1+AJ8a2OfrAP4dwMsAXAYT9KfLHterlAs1g6mrJNMo48o6H7PUg48dpE1qurMVNVMu6wAuzW1fAuC5Ift8TVV/q6o/AfAETF6dUsLUVZJplHFlxbM6nc3ZuKL0VMjaLBZWJaA/CGCniFwmIlsB7AewOrDPVwG8FQBEZDtMCuapJg+UKARV0ig+jsN2IZbZmT4ZuUi0qr4kIh8DcBJmxMvdqvqYiNwG0/Rf7f3tHSLyOID/A/BXqvrfNg+cyFdliwIPLsDgwyLGrlRZGJrGk3ZxLqKWpTYOu0wKqwvZwOJcRJ5IIc1QNaWU4igf2xjQKVyDV5cWrjabznfHPg573Po0Ka8uZAMDOoWphSX0bBTPin0UzDgTq6h5DOgUHm1nCT0bwSn2NEMKKSWfsVOUwpQP4pmGFyZJsXhWXez0tY+dohSfFhYmiT3fbUPsKSXfMaBTmLIWel7DhcAYnMYXY0opqIlgRTUBbN9Yy4Um1uLCJFyMI2x13z8fF+RA3QUubGAOnWrhwiQ0QhMTl3zsEyjLoTOgU7hU+3Pmg9uUtCaCsY8d4+wUpTglVt2xai43qJyvRU0MoQytY5wBnSgAVSc52ZgMFaomgnFoHeMM6EQBqDrJqanJUDG08psIxnVG7Th5DYt6S23fOMqFqLqqiw03sSixjyM7JuVqlJLN1xAc5ULRSaxDtGoHXxMdgT6O7AiNzdeQnaIUlxYKc/mmavqgiTQD67HU5+o1ZECnsLRUmMs3VXO5TczUDG1kh4+cvYZFuRjbN+bQaWL5maHZreEZoimLKYfuiqscOlvoFJ4WCnOlLMZ6LJOoM0rF1WvITlEKTz7Nkmm4dC6lzef1TtkpSvHIB/PFRTP/enGxP6dOUYwjdynUlZcudH0ARGMRMQW48i3yLP0yPc0WOja3LrPZooD71mUoQh3pw5QLhcnlOHTPx8BzHHl9Pr+GTLlQfFwV5gpgDLzt1mUK6ZzQarhkGNBTNXhlxtzzaIGMgbc5BjqV4l+hjvQJN+Xi+WWv17g4xOQCGGFjc4SGz6mIVMSXcgngstdbgbQyvRXAGPgmW5eD6ZVhwRzwv7MwGUUzjmzfJp4pans9ycF/H+PsQ860nFzZaxfZZ2fYbMeiao6djuujTQdKZoqGF9BV7QWk5eX+x8meZ3m53uP6qNvtf/0CDz6tKGtMXH216sJCVJ+dTmd48B4M6iwL0K6ygB5mysXGZa/PqYimOzCzc8vjpJzRisbALyyYv995p3+fnRqK0iiq4XUWJqMo0udvAPYCeALAGQC3lOx3PQAFsGfUY3rZQvcxFdH0VYPtlFUKhqVWfPzs1FTUQmd6xS3USbkAuADAkwAuB7AVwCMAdg3Z71UA7gdwympAbyOH7ksqwta5TvolEVmOuHE+fXYawKqLfqob0K8BcDK3fRDAwSH73QHgTwF813oL3Vau28dWls2rkbLtQSn1L0zCx89OA1wt4UbF6gb06wF8Lrf9fgCfHtjnKgD/1vu9MKADmAewBmBtZmam3lk13Vr0ORXhuuXn82vjA74+1KKygF6lONewnsbf9fKIyBYAhwF8aNQDqepRAEcBM7GownOXHFXDU799LfqkBR2YbY59zr8WR46cn1Tj2YQaZ3z97FByRs4UFZFrAKyo6jt72wcBQFU/2du+CCbH/j+9f/J7AH4B4FpVLZwK6m1xLlV/ZqBmwTwrFXv48ObtNo9N1cwwyXS7DFZ5Pn12KFp1Z4o+CGCniFwmIlsB7Aewmv1RVV9U1e2qOquqszCdoqXB3Guuij4NU9TyW1xsv+VXdKUwokGQFJ8+O5SkkSkXVX1JRD4G4CTMiJe7VfUxEbkNJpezWv4IVMvKSn9LLwvqLoL5sCsFgGkXIk9UWuBCVU8AODFw3ycK9n1L/cOiPq5bfswREwUhzJmi1L6Vlf6WeBbUWRCNxuRDPXUfjsEGLkFH1Y1zpcAOQhrCh+XxfDgGW8Kth07+Yr11KuBDPXUfjqGO+Oqhk7/U4yJn5JwPiy/7cAy2MOVCzeIkJCoxMzO8ddzE8nghHYMtbKFT8wJY1Yfc8GHxZR+OwRYGdGoeJyFRAR8WX/bhGGxhp2gdHMmxmW/lCogiU9Ypyhz6pDiSYzhOQiJyJq6A3laLOT+SA9jcCk29pe5DuQKiBMUT0NtsMXMkx2iuyxUQJSiOTlEXY585koOIPBNHQM+XlT1yxBRosN0J1/ZIjsHH5YgRIhoQR0AH6rWYxw2WgyM5ut3zXyY2gvrKSv/jZs/vovO1jS8WfnkRTSSegD5pi3mSYNnmwhM+TaVv44vFpy8vsirWiodOFS02avu2e/fuxhZNnXiR3rqL+za9UHXZ87heUb6NhZC52HIyjh1TnZrq/0hPTZn7qRxKFomOI6Crqi4v9/+nz4LB8nL5v/MhWFbR7fYfo4vja+O1CuX9oFo6nf63OLt1OtX+/bFjZl8R8zOlL4KygB7XTFGdcBy6er74cT7NknE1RLKN18r394Nq27JleLZQxLzdZQbrmQOmFkss0/dHSad87iRjnyfNvbclH8zb6ICtcix5TR+D7+8HNaKosmGViodLS/3BHDDbS0v1jyt4RU1327fGUy6TCCVnO2k6qUnMoVOD6uTQRYana0TsH7cPUJJyiWem6CRCqTviw1T6Nl6rUN4Pqi1LjSwtmYUlZmZM+doqKZOY65nXFVcOfVKT5t5T1MZrxfeDSjCHnkoOfVKsO1JdG68V3w8q0VQ98xjHwaedciGiIM3N1WuND7byn37abGePHSq20IkoObGOlGFAp2YN9slwuCF56Jlnxrs/FAzo1BzWYaFA1BkH7zMGdGqGelREjGiEQ4fMyJi8qSlzf8jYKZoaW0MCU13FiUMsg1RnHLzPOA49JU0t01cWxFKqw8KFwsmB2uPQRWSviDwhImdE5JYhf79ZRB4XkUdF5Fsi0ql70NSwplIiZXnylOqwMMVE8HAse1FNgOwG4AIATwK4HMBWAI8A2DWwz1sBTPV+/yiAe0c9rhe1XFJTtzRtWa2VhQVzS6kOSxOvZ9k2ec1VTXfUKZ8rItcAWFHVd/a2D/a+CD5ZsP9VAD6tqn9Y9rhMuThSlhKpkv/Nt0QzWZ781lvTS0FMmmJiuiZ4s7PDa8p0OsDZs/aet27K5WIAz+a213v3FbkRwDcKDmReRNZEZG1jY6PCU1OjhqVEDhw438CoMsSwbO3WlZX+DtBs31gD1KQpJqZrJuZTisPLsexFTffsBuDdAD6X234/gE8V7HsDgFMAXj7qcZlysaToMr4oPVKULil7fK4o1MzyhXwdx+LbsnV1V12aFOosQQfgGgAnc9sHARwcst/bAfwIwGtHPaYyoNsxqm56/u/dbn9QHzeYp5InL1O3Tr0PywoGxFUALeJjDr1KQL8QwFMALsP5TtE/GNjnKpiO052jHi+7MaA3rGqwzQeNSQKKD4tt+GTSjk220Mfm48IWLtY2rRXQzb/HPgA/7gXtpd59twG4tvf7NwH8DMDp3m111GMyoFswTpCoE1A4OqMeXulMxLcWuiu1A7qNGwO6JVVa3Qwo7vFKZ2y+5dBdKQvonPofEy0YdTE49Z5Lvbnnw7KCgYl1un6TOPU/FlkwP3LkfKAe3B4MFvmAMmybiLxTNg6dLfRYTNLqLgvmDO5EwWELPTaTBGbOWiQKBheJTsm4CywrZy0SxSLOlAvTB9WlWsecKELxtdBdLYM22JINqWVbVp+FiIIRV0B3lT4IfS3NouGOIX0pEVFkAT1raS4umiC+ZUv5sL0mhJ6DHhzu2O2ef/0Y1ImCEucoF9V2l0HLB8VMSDlojnIhCkbZKJf4Arqr4Nr2l0jT2JFMBMDUWPd5Nmo6wxZdpQ9iyEGPO9yRKELHjwPz82YlIlXzc37eg7VCK4oroBfNllxctFejhDloIjscjBxbWgLOneu/79w5c38I4huH3nbRIxa6Imqeo34dL5eVG0N8AR1oP33AynlEzcmPHAM2F5qz2L8zMzN84eeZGStP17g4A7oLbX+JsBOTYuVw9vKhQyZnnk+7TE2Z+0MQVw49FaFPZCIaxdHs5bk54OhRoNMxT9XpmG2fRrmUYUAPTdWJTOyMpZA5HDk2NwecPWvGN5w9G04wBxjQw1M0G/bKK4Hbb+/vQKraYg+5Dg3FhyPHJsaAHqJhl6OnTwM33zx+6QGmb8g3LoYfR4KdoiEadjl65ZXjdyA5HE1AVIojxybCFnpoii5HT5/u36/Kh99FMTPyny8pOM5eHhsDemiGXY7efrtpoedVzTWyFjrlMQUXNAb0EK2s9M+gu/lm00LPWuwLC/0dSGWBPYY6NNSM0EtBEwN6sPK5xXyL/dZbzf0LC+Z+oLiFxdEElMcUXPAY0GOQtdgB05K6807z+/JyeQuLowloEFNwQYuvHnrqJqkHzzICzYjhdQx9sZYEpFMPnSZrYXE0QX0rK8CBA/2diQcOhNWZyBRc8BjQXbE1NIydnO1TBe67z6S6sqB+4IDZvu++cF57puCCx4lFLtiq9TzYwspPFAJ42WzT1VcDDzxggnjWh5Hd79o4qSBO6AlapRa6iOwVkSdE5IyI3DLk7y8XkXt7f39ARGabPtBo2BwaxhaWGyLAHXeYkUV5Cwvmfpev+yTjypmCC9bIFrqIXADgLgB/AmAdwIMisqqqj+d2uxHAL1X190VkP4C/A/BnNg44eLZrPbOFRRmWdkiPqpbeAFwD4GRu+yCAgwP7nARwTe/3CwH8HL0RNEW33bt3a9K63WzKj7l1u66PiCbV7aouLPS/n9ltYcHte9vtqi4u9h/T4iI/bwEDsKYFcbVKyuViAM/mttd79w3dR1VfAvAigG2DDyQi8yKyJiJrGxsbVb9z4sOOy/g88ID5ubBwfrZu/n5XOK48KVUC+rB3fjDyVNkHqnpUVfeo6p4dO3ZUOb74cGhYfESAvXv7c+ZZTn3vXrfBk42HpFQZ5bIO4NLc9iUAnivYZ11ELgRwEYBfNHKEsSnquATYcRmyYX0XrjtEOeopOVUC+oMAdorIZQD+C8B+AO8b2GcVwAcBfB/A9QC+3cv10DDsuIyTb6ND2HhITqWp/yKyD8AdAC4AcLeqHhKR22CS86si8goA9wC4CqZlvl9Vnyp7TE79J2pJDCUJ6HfKpv5XmlikqicAnBi47xO5338N4N11DpKILPHtyoGs4dR/IqJIMKATEUWCAZ2IKBIM6EREkWBAJyKKBAM6EVEkGNCJiCLhbE1REdkA8PQE/3Q7TDXHlPCc08BzTkPdc+6o6tBiWM4C+qREZK1ollSseM5p4DmnweY5M+VCRBQJBnQiokiEGNCPuj4AB3jOaeA5p8HaOQeXQyciouFCbKETEdEQDOhERJHwNqCLyF4ReUJEzojILUP+/nIRubf39wdEZLb9o2xOhfO9WUQeF5FHReRbItJxcZxNG3Xeuf2uFxEVkaCHuFU5XxF5T++9fkxE/rntY2xahc/2jIh8R0Qe7n2+97k4ziaJyN0i8ryI/LDg7yIid/Zek0dF5I2NPLGqeneDWRnpSQCXA9gK4BEAuwb2+QsAn+39vh/Ava6P2/L5vhXAVO/3j4Z8vuOcd2+/VwG4H8ApAHtcH7fl93kngIcBvKa3/VrXx93COR8F8NHe77sAnHV93A2c9x8BeCOAHxb8fR+AbwAQAG8G8EATz+trC/1NAM6o6lOq+hsAXwZw3cA+1wH4Qu/3fwXwNpFgl2IZeb6q+h1VPdfbPAWzWHfoqrzPAPA3AP4ewK/bPDgLqpzvhwHcpaq/BABVfb7lY2xalXNWAK/u/X4RNi9CHxxVvR9mOc4i1wH4ohqnAEyLyOvqPq+vAf1iAM/mttd79w3dR1VfAvAigG2tHF3zqpxv3o0w3+6hG3neInIVgEtV9ettHpglVd7nKwBcISLfE5FTIrK3taOzo8o5rwC4QUTWYZa6/Hg7h+bUuP/nK6m0pqgDw1rag+Mrq+wTisrnIiI3ANgD4I+tHlE7Ss9bRLYAOAzgQ20dkGVV3ucLYdIub4G5CvsPEXmDqr5g+dhsqXLO7wXweVX9BxG5BsA9vXPu2j88Z6zEL19b6OsALs1tX4LNl2G/20dELoS5VCu7xPFZlfOFiLwdwBKAa1X1f1s6NptGnferALwBwHdF5CxMrnE14I7Rqp/rr6nqb1X1JwCegAnwoapyzjcC+AoAqOr3AbwCpoBVzCr9nx+XrwH9QQA7ReQyEdkK0+m5OrDPKoAP9n6/HsC3tdfbEKCR59tLPfwTTDAPPa+aKT1vVX1RVber6qyqzsL0HVyrqmtuDre2Kp/rr8J0gENEtsOkYJ5q9SibVeWcnwHwNgAQkdfDBPSNVo+yfasAPtAb7fJmAC+q6k9rP6rr3uCSXuJ9AH4M00O+1LvvNpj/0IB50/8FwBkA/wngctfHbPl8vwngZwBO926rro+5jfMe2Pe7CHiUS8X3WQDcDuBxAD8AsN/1MbdwzrsAfA9mBMxpAO9wfcwNnPOXAPwUwG9hWuM3AvgIgI/k3ue7eq/JD5r6XHPqPxFRJHxNuRAR0ZgY0ImIIsGATkQUCQZ0IqJIMKATEUWCAZ2IKBIM6EREkfh/9e5pUEP0wecAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "util.plot_points(Xb[:,1:3],Yb==1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like dataset B is linearly separable, while dataset A clearly is not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x221363d6dd8>]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2de3wU9dX/PyeBCBEkCmgRSAISrCC3wiNaUJ5SQUUB79V6aS02tiqk0j4FDJog8mgvgojWCtbHn4IVq2gBUW5ViBYMoUIwgBgQA4IQgUQ0QC57fn/MDsxu9jK7O/c579drXsnszs6c7+zsme+cKzEzBEEQBPeTZrcAgiAIgjGIQhcEQfAIotAFQRA8gih0QRAEjyAKXRAEwSO0sOvAHTp04NzcXLsOLwiC4Eo2btz4NTN3jPSebQo9NzcXZWVldh1eEATBlRDRF9HeE5OLIAiCRxCFLgiC4BFEoQuCIHgEUeiCIAgeQRS6IAiCR4ir0InoBSI6SESfRHmfiOgpIqokonIi+oHxYgqCIAjx0DNDfxHAlTHevwpAXnDJB/Bs6mIJgiAIiRJXoTPzWgCHY2wyFsBLrLAeQBYRdTJKwHAqD36LJ1Z8iuMNTWYdQhAEwZUYYUPvDGCPZn1v8LVmEFE+EZURUVl1dXVSB1u97QDm/KsSo54qQennse4zgpYFC4DcXCAtTfm7YIHdEgmCYDRGKHSK8FrErhnMPJeZBzHzoI4dI2auxuWeYefhpV9chPrGAG5+bh0K39yCo8cbktqXX1iwAMjPB774AmBW/ubni1IXBK9hhELfC6CrZr0LgH0G7Dcql/XsiBUPXIZxQ7vh76VVGDFzLVZtPWDmIV1NYSFQVxf6Wl2d8rogCN7BCIW+GMCdwWiXiwHUMvN+A/Ybk8yMFnjoml5YdO8QZGW2xN0vleG+V/6D6qMnzD60I4llUqmqivyZaK8LguBO9IQt/h3AOgDnE9FeIhpHRL8iol8FN1kGYBeASgDzANxrmrQR6N81C4vvH4rfjuiJlRUHcPnMNfhH2R4ws2/sxvFMKtnZkT8X7XVBENwJ2dUketCgQWx0tcXKg99iyqJybNh9BOe16YDSZ/rg268yT76fmQnMnQvcdpuhh7Wd3FxFiYeTkwPs3n1K4WvNLl49F4LyfRcWKk9g2dnAjBnyPXsJItrIzIMivuclhQ4AgQBjwUdf4KHXP0WAGTUf9MTRslyAlYcRVcl5ibQ0ZWYeDhEQCCj/y4/cH8jN2/v4SqGrtGx3DGeO+ASZPQ7ixP52OPROXzRUn3FKyTErGs8DxJuhC/7BjGtBJgPOIpZC92wtl85ntkb1G4NQ/c8BaHHGMXT62QfIunQ7snObFGX+wANAcbHdYhrCjBnKLExLZqbyuuAvjHaAS8iru/CsQleUHKFu+7nY9/wwfLe1M9r9cCc63FGC0onTgNmzgZqayLYKnTjF6XrbbcojdU6O8tCRkyOP2H7FaAe4hLy6C88qdK2S4xMt0aakK+5auBqtDuzEzaf9Fwp/+yyOPvbHpM0uTpu53Hab8kgdCCh/vajMnXIDdTJGP61JyKvLYGZbloEDB7KlBALMAH/X8jR+ZPjd3G3yUh48YxWvqPgq7kfnz2fOyWEmUv6q64oqD11yckweh0+ZP585MzP0XGdmKq8LoUS6XpPFquvcSJm9DoAyjqJX/aHQAwHmgoKQK/Lj3zzEI2e+zzmTlvK9CzbywW+OR/xoNEUS6SIHlAtSMB65gdqDFTdSuVknRiyF7lmTy0lUB+js2UBBgWKTKChA/yenY8kXb+F3I5snJGmJZkNMT498OEnWMQd59LcHM/wz4aazggKx0xuF9xU6EZCVpVw1s2Yp67NmAQUFyMhqh/uH52FZwaXoeU4b/M/r5bjjb6WoOnTq6oqmMJqa/BNZ4gTbtWS72oeR/plIvqdDhyJvKzfrJIg2dTd7scWGHmO9qSnAL63bzb0ffpfPn7qM567ZyY1NgZiP+n6w+znlcdgpcgipEe33JOY0/SCGycWziUXJsr/2GB566xOs2nYQ/bq0w2Wt+uLhgjN8m3nnpKQlSXBxP9GymsPx028sUXyZKZoKzIyl5ftRvLgCtccaMLT9eXj/mR6o2p3uO0Wip6yAIOgl2gShfXugTRu5WevBl5miqUBEGN3vXKyaOAxj+3fG+9WV6H5fCdbvPOzZGO9oiO1aMJJocfKzZ3s/j8IKRKHH4MzTM/DEzf3w8jj/dkiSsgKCkUhWs7mIQtfBpXlKh6S7NR2SVvqkQ5L8AL2LXdFLfshqtguxoSfIpj01mPxGObZ/dRRX9+mE4jG90bHtaXaLJQgJIWV23Ys4RZOFw0rsBtfrGwN4bs1OzPlXJVpnpGPq1RfgxoFdQB4pxyt4HydFLwmJIU7RZCguVjJM1RuemnFaXIyMFmkY/+PYCUluwAkJQ4I9SOatNxGFHglmpbTu7NmnlLpaPkBTcrfH2W2wMP8STL/2QmzaU4MrnlyL50t2obHJ+fF8TqsWKViLk6KXZGJhINEyjsxeLM8UTZQIBb24oKB5xmmQfTV1PO7FUs6ZtJRHzynhrftqLRY4MaTYlb9xSuatU+RwE5BM0SRhVqYNKoFAzPrpzIy3tygJSTV1DbhnWHeMH56HVi2jVPKyEUkYEpyQeSu2/MQRG3oyqGYWLVqbegSICNf0PRcrH1ASkp55bydGPVWC0s8Pmyxs4jjpkVuwByeED1pty7fDvGPpMaNN3c1eHG1y0ZpbVDNL+LoO1nx6kIc8vppzJi3lBxeVc+2xepMF14886gpOwErTnx3XvBnHhO8bXCRDUVGo8laVelFRQrv57kQDT19SkVCHJKvwQ7VIwdlYqWTt8BuZccxYCl1s6LHgyHHoybB5Tw0mqQlJfTuheLQkJAkCYJ0t3w6/kRnHlMQih1DfGMDctTvx1GolIemha3rhhh90loQkQbAAOxywZhxTnKIOIaNFWkiHpN/9YzPufKEUew67KyFJENyIHYXmrD6mKHQb0CYkfVxVg5GzlISkpoA9T0uC4AfsKDRn9THF5GIz+2uPYeqbn2D1dqVD0uM39MUFnc6wWyxBEByKmFwcTKd2rfH8zwZhzq0DsPfIMYye8wH+tHw7jjc02S2aICSNpPPbgy6FTkRXEtGnRFRJRJMjvJ9NRO8R0cdEVE5Eo4wX1buEd0hSE5I27HZeQpIgxEPqBNlHXIVOROkAngFwFYBeAG4lol5hm00F8BozDwBwC4C/GC2oaYSbnGwyQQHNOyTd9Nd1mPqWOzokyYxMUCksDK2zDijrhYX2yOMn9MzQLwJQycy7mLkewKsAxoZtwwBUw287APuME9FEYpTItRNth6RXPnJ+hySZkQlapDSvfehR6J0B7NGs7w2+pqUYwO1EtBfAMgDjI+2IiPKJqIyIyqqrq5MQ10BYX4lcu8jMaIGp1/TConuHICuzJX75Uhnue+U/qD56wla5IiEzMkGL1AmyDz0KPVLWS7i2uxXAi8zcBcAoAC8TUbN9M/NcZh7EzIM6duyYuLRGQgTMmgUUFChKPC1N+VtQoLzukGSf/l2zsGT8UPxuZE+srDiAy2euwT/K9sCu6KRIyIxM0CKNxe1Dj0LfC6CrZr0LmptUxgF4DQCYeR2AVgA6GCGgqahKXYuDlLlKy/TQhCSndUiSGZm9OM1/IY3F7UOPQt8AII+IuhFRBhSn5+KwbaoA/BgAiOgCKArdZpuKDpIokWsn0Tok2Z2QJDMy+3Cq/8IJpXl9SbSqXdoFihllB4CdAAqDrz0CYEzw/14APgSwGcAmACPj7dP2aosGlci1iy+P1PEv/k/pkDTGAR2SpHKjPUjnKf8BqbYYheJixQGqmlnUGXtWlu2RLnpgZiwtVzok1R5rwK+GnYf7h/dwZIckwRyk85T/8F+1xUTK3hpYItcujnxXj0ff3oY3/rMX3Tucjsdv6IuLup1lt1iCBUgLN//hr9T/RGPLw5W3y5Q5EJaQ1BTAzc+tQ+GbW/CNCxKShNQQ/4WgxVsK3crYcgdlmKpoE5L+XlqFkQ5PSBJSRyJKBC3eM7lolbiK0bHlLrC9h3RI6tMJxWOkQ5IgeAF/mVzMji13eIapSj9tQtJWZyYkCYJgLN5T6GbHlrskwxRwfkKSIAjG4i2Frp0tFxQocVuq4jVDqWtxmDLXcjIhaWzvkISkxiaJa0sUp2VlehU5z8nhLYVOpNixtbNldTadlWWs2cVFGaYAkJZGuOOSXKyceBmG9GiPR9/ehuuf/Te27f/GbtFcg1OzMr2GnOfk8Z5TFDA3tjz8KWDWrObrDp2pq6gJSdOWVKCmrgH3DOuO8cPzJCEpDhLzbQ1ynmPjL6coYG5suVVPASaidkha+cAwXDsg2CFpdgk+2nXIbtEcjVSVtAavnWcrzUfenKFbgQcyTFVKPqvGlEVbsPfIMdw2OBuTrvo+zmjV0m6xHIfMHK0h1nmeMUOps19VpVTznDHD2TH3qvlI2y8gMzO1XAH/zdCtwAMZpipqQtI4SUiKiWRlWkO08zxqVHPb+l13AR06ONd5anXzF1HoAgClQ9JDLumQZBeSlWkuqmnijjuA1q2B9u1Dz/OyZc2VY0MDcOiQc52nVpuPxOQiNKO+MYC5a3fiqdWVaJ2RjqlXX4AbB3YBufgpRHA2ekwT0SpLhuMkE5gZZjoxuQgJkdFCEpIEa9FjmtDbActJzlOrzXSi0IWohHdIGvnkGsxbKwlJgvHoMU1EUo6ROOss5yQlWW2mE5OLoIv9tcfw0FufYNW2g+jbpR3+cENfXNDpDLvFEjyCXtPEggWnolzOOgs4ehSorz/1fkaGYpZp0FSOTjWqxGmIycXLWFTGt1O71ph35yDMuXUAvjxyDKPnfIA/Ld+O4w1NphxP8Bd6TRPaXqVffw288ELo7Ldt21BlDpgbVeI0RKG7mUSbeaSImpC0auIwjO0fTEh6qgSlnx825Xh2IrVErCVZ00R4M+rDUS5FJ9nVzUQUuluxsYxvSIekxlMdko56pEOS1BKxh3DlnIyJJJrjVK9D1e2IDd3NWNHMIw519Y14YsUO/N+Hn+Pstq0w/doLMaLXOZYc2ywkI9S9mJGZ6TT81yTaTzArdgGVQCAxZW5QCYNNe2owWe2Q1LcTike7t0NStHhnIuX0Cs5G6zh1Q3mARBGnqFdJtYyvgTb4/toOSRVKh6TXXNohye+P7W7HCNONWxGFrsWBjZ+jkmozDxNs8OEdkn4fJSHJ6Q5HqdniT5x+XeqCmW1ZBg4cyI6iqIi5oIA5EFDWAwFlvajITqlik6rM6vaK+lYW7f5SoKkpwC+t2829H36Xz5+6jOeu2ckNjU08fz5zZmboITMzmefPT/mQhjJ/PnNODjOR8tdp8gnG4pbrkpkZQBlH0atiQwfc3bQiVRt4qjb4OIQnJJXP64svypsnJInDUbATNznCxYYeDxc1fm5GKmV8LWilF56QxCM/QNal24H00IQkv8QJC87EK001RKGruKzxc8qY1VA7gh9Cm5BEX3RGux/uxLl3leC0Lqc6JInDUbATrzjCRaGruLDxc0qY0UovTtTMmadnYMbYfqh56yJQegDfu209zhq5BadnNfja4egJZ1yCOG3MnnGERzOum704yimqdQ6qTsHwda8SPrZkx5rAOZw/nzmnewOfObyCc/5nKfcpXMUrKr5KYRDuxU3OOKNw6pjNcoQbvV/EcIrqUr4ArgTwKYBKAJOjbHMzgK0AKgC8Em+fjlLozO6McnEaSUTNfFx1hK+YtYZzJi3lexds5IPfHLdQYPPQ+yPOyQk9XeqSk2OdrFbjpzGbcfOKpdDjRrkQUTqAHQBGANgLYAOAW5l5q2abPACvARjOzEeI6GxmPhhrv46KclFJNWJESCpqxmsdkhJJP/djVqqfxuzEjkUXAahk5l3MXA/gVQBjw7b5JYBnmPkIAMRT5o7FQ42fbSFJP4TXOiQl0hjYK864RPDTmK2OntGj0DsD2KNZ3xt8TUtPAD2J6EMiWk9EV0baERHlE1EZEZVVV1cnJ7HgTAyImgnvkHTFk2vxfIn7OiQl8iP2jDMuAaJ1Hvr2W/udo0Zj9c1Lj0KPNE0N/3W2AJAH4L8B3ArgeSLKavYh5rnMPIiZB3Xs2DFRWQUnY1DUTFoa4Y6Lc7By4mUY0qM9Hn17G65/9t/Yuu8bkwdgHIn8iK1uUeYE1DG3bx/6+qFD3itTbPkNO5pxXV0AXAJguWZ9CoApYdv8FcDPNeurAfxXrP06zikqGINRUTPMHAgEeMnmL3ng9BV83pS3+Y/vbuNj9Y0pCmg+To3icBp+cY5aGeWiZ4a+AUAeEXUjogwAtwBYHLbNWwB+BABE1AGKCWZXivcawY0Y6IcgIlzT91ysfEDTIWm28zskxZt1Oy0G2y68kp0ZDyurP+qq5UJEowA8CSAdwAvMPIOIHoFyp1hMSjjCE1DCG5sAzGDmV2Pt05FRLoKjKfmsGlMWbcHeI8dw2+BsTLrq+zijVUu7xUoIPzRg0Iub6qc4CWlwIXiGuvpGzFyxAy+4tEOSKLFTyM0tOaQ4l+AZMjNaYOo1vbDo3iHIymyJX75UhvsW/AfVR0/YLZou/GJm0GNW8qND2GxEoQuupH/XLCy+P9ghaaumQ5LBmSlG27v9EIOdSJNtP3cXMgNR6IJryXj0Edy/+BksmzD0VIekSfNR9fBjhuw/EcWkFz/EnSeSWCUYiyh0wZ3wqRZ6PR57CAt/eTGmN2zDpsbWGHmiN+at3ZlyQpIZiskPZga/mJWciDhFBfeizU4Nsr/g93howI0nOyT94Ya+uKBT8w5JevBTzREjEcevuYhTVPAmEZqSdJr1eEiHpNFzPsCflm/H8YamKDuJjh/s3WbgB7OSUxGFLriXKMXACDjZIUmbkPTRrkMRdxMNUUzJ4QezklMRhS64Ex3FwM48PQNP3NwPL4+7CPVNAfxk7noUvrkF3xxv0HUIUUzJ46TolVQjldyU2Ss2dMG9FBcrjlG1GJiq5LOyTra9U3F7QpKQHKkmLzkx+UkyRQXvkmBTks17ajDpjXJs/+ooru7TCcVjeqNj29MsEFSwg1QdtE508IpTVPAuCRYD69c1C0vGBxOStmkSkmya2CSKnsd/N5kIzCbVEEq3hWCKQhd8R8t0pUPSOwWX4vxz2ioJSS7okKQn0cmMZCg3k2qkktsinUShC77lvI5t8Gr+xXg02CFp5JNrMG+tczsk6Ul0MjIZygsz/VQjlVL5vC3nL1qhdLMXaXAhOIl9NXU87sVSzpm0lEfPKeGKL2vtFqkZRJEbQhAlto0evNSkI9UGE8l83szzhxgNLsQpKriXBB2i8XfHWFq+H9OWVKCmrgH3DOuO8cPz0KplugHCpo4eB51RTjwnOgPdhJnnT5yigvcoLg5tPq2GLIaFKyYCEWF0P6VD0rUDkk9IMgs9j/9GJUO5zRnoNOw6f6LQBfehKcx1UqmrSUY1NZELsCTAmadn4M83JZ+QZBZ6Ep2MSoZymzPQadh2/qLZYsxexIYupEQgwFxQEGqkLChIqSl1JL470cDTl1Rwt8lLefCMVbyi4itD9+9UvGRDtwO7bOgyQxfcSYTCXCczRg0kYoekV9zTISlZpOxBalEqdp0/cYoK7iRC6VwUFJii1FXqGwOYu3YnnlpdidYZ6Si8+gLcNLALyKTjCfbhxJR/FXGKCt5CR2EuM8hooSQkLSu49GSHpNv/9pFjE5K8EEduF27tuiQKXXAfREoBLu2MfNYsZT0ry7QZukqPs9tgYf4lePTaC7F5T60jE5IkYzQ13BrlIyYXwb0YHIeezLH31x7DQ299crJD0uPX90Wvc5PrkGQkEkeeGk4+f2JyEbxJgoW5DEMTA9+pXWvMu2Mgnq4vx769X2PM08l3SDISM2eYfjDluLW5iSh0QUiECDHwNHEirpn1IFYe/xDXDjjXEQlJZsVB+8WU49YoH3ebXOx85PYCcv6SI06ETcln1ZiyaAv2HjmG2wZnY9JV38cZrVpaKqJZURpONkX4BW+aXExI/fYVcv6SJ04M/KV5HbHigctw99Bu+HtpFUbMXIMVFV9ZKqJRM8xw80okZQ4431noF9yp0E1O/W72eZueYkzD7PPnddTzpUU9j8FzdzIh6dc/xJmZGch/eSPuW2BtQlKqfT0jmVeiPcBJSQCHEC2F1Owl5dR/s1K/i4pC96Mep6gotf06DYtS5z2H9ryp50tdHzyYecKEZtdO/cPFPGf1Ds57cBn3LV7Or22o4oALznNOTujloS3FKyUB7AMxUv/dq9CZlR+M9spK9UcS68fqRWVn9PnzC5Fu+hMmKAo9xrXz2YGjfOOzH3LOpKV827z1/MXX39k6jHhEq60OpFZfXEiNlBU6gCsBfAqgEsDkGNvdCIABDIq3T8fO0J06cw0/vlfH6RYifR86zmlTU4BfWrebez/8Lp8/dRnPXbOTGxqbLBZeH9Fm6Dk5dkvmb1JS6ADSAewE0B1ABoDNAHpF2K4tgLUA1puu0M2eSTtt5mq0GSiV82f0jcVr6Lx23NAhSSouOpNYCl2PU/QiAJXMvIuZ6wG8CmBshO2mA/gjgOOJW/ITxMzUb47h8LIDNsGBmez5k8iY2CRw7XRq1xrz7hyEObcOwJdHjjkmIUmLW2OxfU00Ta8uUMwoz2vW7wDwdNg2AwC8Efz/fUSZoQPIB1AGoCw7Ozv1W5WZZggn2dDNNC/FWo8mg5POjVNI4fwc/vYET1y4iXMmLeUf/ek9Xr/zawsFF9wGUukpSkQ3AbiCme8Ort8B4CJmHh9cTwPwLwA/Z+bdRPQ+gN8xc8ysIcfWcikuVma+6sxVnXVlZdk7E2VWgoFVAgHrk4C0TwcqJpesdRUpXjsln1XjwTe3YM9h+xKSBOcTK7FIj0K/BEAxM18RXJ8CAMz8WHC9HRQb+7fBj3wPwGEAY2IpdccqdMB5GZROUqROuLE4mRSvnbr6RsxcsQMvfPg5zm7bCtOvvRAjep1jgqCCW0k1U3QDgDwi6kZEGQBuAbBYfZOZa5m5AzPnMnMuFKdoTGXueOwq+hQJrTK3sPZ3TFm02OlfcCIpXjtqQtKbPuuQJBhDXIXOzI0A7gewHMA2AK8xcwURPUJEY8wW0PfYXPv7JE66sfiAfl2zsGT8UPxuZE+srDiAy2euwT/K9iDeE7Xgb9xdnMtPOMEM5FT/gsepPPgtpiwqx4bdRzC0Rwf873V9kN0+M/4HBU/izeJcfsMJZqDi4lC7vfq0IMrcVLQdkjbtqXFkh6REsbumut3HNwuZoQvm4YSnCo/h1A5JiWB3A2a7j58qKUW5mIUodI8j5hnTYGa8vWU/ihdXoKauAfcM647xw/PQqmW63aLpwu6a6nYfP1XE5CJYixnZrcJJiAjX9D0XKx8YhmsHdFY6JD1VgtLPD9stmi7sbsBs9/HNRBS6YDzaSJzZsxVDpRodI0lIhnHm6Rn480398PK4i1DfGMDNz61D4Ztb8M3xBrtFi4lZ7fHccnwzEYUumEOcrj6CcYR3SBo5c63lHZISwe4GzHYf30xEoQvmIElIlnKyQ1IwIcmODkl6sbvol93HNxNxiqaKRHI0JzwJadas5ut+P0cmUt8YwNy1O/HU6kq0zkhH4dUX4KaBXUByzj2BOEXNQsrJRsYp2a0+JaNFGu4fnodlBZei5zlt8PvXy3H73z5C1aG6+B8WXE0LuwUwBStmzdpIDqD5LNTvM/Xi4tBzoCp1P58Ti1ETkhaUVuEP72zHyCfX4LcjzsddQ3LRIl3mcl7EeyYXK+OfnVQFURBi4IWEJEHBPyYXq+OfJZJDcAlqh6SnfzoA+2qc2SFJSB1vKXSr458lkkNwERETkmaX4KNdh+wWTTAIbyl0IPlZc7gSjqeU7Sonm6icbpXBCeP0KCEJSU0B/GTuejzogoQkIT7eU+jJzJqTiVaxI5LDCVE1VsjghHH6AG1C0qulVRgxc42lCUlerXhoK9GajZq9DBw4MNVeqc1JplFvqs2PjW5UHes4djdptkIGJ4zTh2yqOsJXzFrDOZOW8q/nl/GBb46Zerz585kzM0P7nmdmKq8LsUGMJtHeUujMzEVFoT98VSEUFUX/jFZpqIsTlYcT5LRCBieM04fUNzbxnNU7OK9wGfctXs6vbajigEnnPCcn9OtVl5wc/fuYP1/Znkj565ebQSyF7r2wRSC5OHR2SfNjJ8hphQxOGKdPsaJDUlpaZCsokfJVx8PtNc1TwT9hiyqJdvdJxu5uB06Q0woZnDBOH6MmJE03sUNSqhUPCwtDlTmgrBcWpiaX64k2dTd7Mc3kkihusdk6QU6xofuOfTV1PO7FUs6ZtJRHzynhii9rDdlvqjZ0osgmGyJDxHM0iGFy8WbqfyJEi1YBnFV3xAlyWiGDE8YpnERNSFpavh/TllRgzNMfGNIhSTWLFBYqjSWys5XytXrNJdnZkbsOeaGmeSp404aeDMnY3e3ACXJaIYMTximEUFNXj0ff3obXN+5F9w6n47Hr+2Bw9/a2yCI2dD/Z0JMhUbu7XThBTitkcMI4hRCyMpWEpPnjBqMhoCQk2dUhyYia5l6Mg5cZuiAICVNX34iZK3bghQ8/x9ltW2H6tRdiRK9z7BZLN26e4ceaoYtCFwQhaTbvqcGkN8qx/aujuLpvJxSP7o2ObU+zW6y45OZGtsHn5AC7d1stTWKIyUWwlvBJgoQbepZ+XbOwZPxQ/G5kT6ysOIDLZ67Ba2V7YNdEUS9VVYm97hZEoQvGInVYfEfLdKVD0ju/uRTnn9PWFR2SUo2Ddyqi0AXjYIvr0QuO4ryObfBq/sV49NoLsXlPrSkJSUYxY4ZiM9eSmam87mbEhu5XzAoL1CpxFa93cZIQy2ZoOyT16dwOj9/QB73PbWe3WCEsWJB8HLydiFNUCMWINn2xlJif6rBY2fLQZTAz3t6yH8WLK3CkrgH3XNYdE36cWkKSYIBTlIiuJKJPiaiSiCZHeH8iEW0lonIiWk1EOakKLZiEEWaRWHZyP9VhERNTTNQOSasmDsN1AzrjL+8rHZJKPz9st2iG4bhY9mg1AdQFQDqAnQC6AwxoT4sAAA3jSURBVMgAsBlAr7BtfgQgM/j/rwEsjLdfx9Ry8SOplKeNVWtlwgRl8VMdllRL/VpVT98BlOyo5qF/WM05k5byg4vKufZYvd0ipYRdNd2RSvlcIroEQDEzXxFcnxK8ETwWZfsBAJ5m5iGx9ismF5uJZhbRY/+NZSefNs1/JohkTUw+NNdoE5I6tj0N08deiJG9v2e3WElhVyx7LJOLnhn6jQCe16zfAUVhR9v+aQBTo7yXD6AMQFl2dra5tzEhOtFmlU1N8ZuBaPeh/bx2ZumjWWfSM3SfV5XUdki6d/5G3R2SnNTUwq6Kj0ilYxGAmyIo9DlRtr0dwHoAp8Xbr5hcTCSWQtUqjv79I/+Np1Cko5CCEe0LfXweT3ZIenAZ9yl6lxeWxu6Q5LS2dUZ0XUqGVBX6JQCWa9anAJgSYbvLAWwDcHa8fbIodPPQ04JP3UadkSeiUHw+s2xGMi0PtcR60vEJnx04yjc9+2/OmbSUfzpvHX/x9XcRt7NLgUbDiTZ0PQq9BYBdALrhlFO0d9g2A6A4TvPi7U9dRKGbQCLKVquAElUoqSoxr5GsicnnM3QtTU0Bfnndbu798Lt8/tRlPHfNTm5obArZxolNLewwAaWk0JXPYxSAHUGlXRh87REAY4L/rwJwAMCm4LI43j5FoZtEIkoi1WiXWOtCbORJJyKxOiQ5bYZuFykrdDMWUegmomfWLQrFfuRJJyKBQICXbP6SB05fwedNeZv/+O42Plbf6Dgbul3EUuiSKeo1OEZIYXgonQ/D5hwHs5QNiEKkDkmV69q7Ml3fSCT13y9olbmqxMPXw5WFKBTB4Xzw2deY8mY59hw+hp8Ozsbkq76PM1q1tFss24il0KVJtJdIpsFyLGUuyl1wAEPzOmD5by47mZC0etsBVyckmYnM0L1IMopZzC+CCwjpkNSnE4rHuKNDkpFIxyK/kWiDZWYpMiW4gpAOSVvd0yHJKrw9QxcTgn4ScaYKggOoPPgtHly0BaW7D2NIj/Z47Lq+yG6fGf+DLsefTlG7TAhuvokw+6eOueAJAgHGK6VVePyd7WgMBPDbEefjriG5aJHuXeOD/0wudpkQ3NxPU5VVi1frmAueIS2NcPvFOVg58TIM7dEBM5Ztw3V/+Tcq9tXaLZo9RAtQN3sxPbHI6rRqNyfquFl2QQiiTUjqPuVt/sM7SkKS14BvE4usNiG42Q4tUS6CR9AmJHXrcDoev74PBndvb7dYhuFPG7pdytXNdmg32/8FIYxkE5Kc3jzanzZ0bYZkIKD81drUzTyuFjfZoRMNdxQEB6MmJN09tBteLa3CiJlrsKLiq5ifWbAAyM9XOhExK3/z8x3QK1Qn3lTo0TImCwqiZ0ymil03EUHwKuG/mSR+Q5kZLTD1ml54894hODMzA/kvb8R9C/6D6qMnIm5fWAjU1YW+VlenvO4GvGtyAaw3IYgdWhCMwYTfUkNTAHPX7sLs1Z+hdct0FF59AW4a2AWk0QlpaZHvG0TKHM0J+LeWi9UmhOLi0JuG+mQgpgtB0I827BhoXmQuyYlZy/Q03PejHrjywu9hyhtb8PvXy/HPTV/if6/rg5z2pwNQbOaRGj9nZ6cyIOvw9gzd64gTU/AqJgc1hCckTRzRE78Y0g0LX01Dfn6o2SUzE5g71zmOUX9GuXgdMe8IXseCiLH9tcfw0FsVWLXtAPp2aYfHr++Lj987Q6JcBAvRmwkrjljBrVgUMdapXWvMu3Mgnv7pAOyrOYbRT3+AL9tvx/bPmhAIALt3O0uZx0MUuhvRRu3Mnq3MYmbPBvr3B2bODJ2x652tGxBRIAiGYHHEGBHhmr7nYtXEYbhuQGf85f2dGDW7BB/tOmTocaxAFLpb0TavUNm0CZg4MfHaNW6uQSN4DzvCjgFkZWbgzzf1w/xxg9EQCOAnc9fjwTe34JvjDaYczxSi1QQwe5Em0SkSqVZN//6J166ROi6CUwm/9iy8Fr870cDTl1Rwt8lL+aIZK3n5J/stO3Y84NtaLl4l/JFUG9alRa8Tyc01aATBRLQdkkb1+R6Kx/TG2W1b2SqTRLl4kfAol0AAGDhQMbuoJKKU3VyDRjAWCYcNoaEpgOfW7MRTqyvROiNyQpKVSJSLFykuDg1ZnDhRUeaqE2nChFAnUqwbt9tr0AjGIf6UZrRMT8P9w/OwrOBS9DynDX7/ejnu+Fspqg7Vxf+wxYhCdzPajFStE2naNOX1CROU14HoP0qpQSOo6A2H9Sk9zm6DhfmXYPq1F2LTnhqMfHIN5q3dhcYmh9QEAMQp6ikCgVCn5oQJ+pycRUWh76mfKSqyUnrBCVjdGMal7Kup43EvlnLOpKU8ek4JV3xZa9mxIU5Rn5GMk1PspqnjlXMo/hRdMDPe3rIfxYsrUFPXgHuGdcf44Xlo1TLd1OOKDd1vRIpRj+cclVroqaHantWSfOpNtajIVrESRvwpuglPSHrmPSUhab2NCUmi0O3ErOxM+VFai9b2PHCgotTVJ6TFi92j1MWfkhRZmRn4kyYh6Za56zFlkT0JSaLQ7cKsaAL5UVoPkVJyoX9/JdIoPf1UKYZNm4DaWnvPu96Jg00Zml5B2yFp4QZ9HZIMJ5pxXbsAuBLApwAqAUyO8P5pABYG3/8IQG68ffraKWp2dqY4Oe2hqSnUmegEh2Iy14KNGZpeYVPVEb5i1hrOmbSU752/kQ9+c9ywfSOGU1SPMk8HsBNAdwAZADYD6BW2zb0A/hr8/xYAC+Pt19cKndn8aAL5UVpLpO8TUJS8E2SSsg6WU9/YxHNW7+C8B5fx8yW7DNtvqgr9EgDLNetTAEwJ22Y5gEuC/7cA8DWCWajRFt8rdGblB6X98csPzJ1oFWV4PZ3+/Z2j1J3y1OAzPq/+lhsajbsGYil0PTb0zgD2aNb3Bl+LuA0zNwKoBdA+fEdElE9EZURUVl1drePQHkYcl96BCGjX7pTNvKAAaGo6ta5WwLRLtkQjngRDye1wOlqkW+Ou1HOUSN98+NWpZxsw81xmHsTMgzp27KhHPm8ijkvvMW0aMGbMKYdiWhqwcaP9DkWZOPgKPU2i9wLoqlnvAmBflG32ElELAO0AHDZEQi8SLZoAkGgCNzNtWmgyUVqavbPh8IlDeFVOmal7Dj0KfQOAPCLqBuBLKE7Pn4ZtsxjAzwCsA3AjgH8FbT1CNIqLQ3/8qlKXH5i7cVKClkwcfIeu1H8iGgXgSSgRLy8w8wwiegSKcX4xEbUC8DKAAVBm5rcw865Y+5TUf0GwCO3EIdK64Cpipf7rmaGDmZcBWBb22sOa/48DuCkVIQVBMAknPTUIpiKZooIgCB5BFLogCIJHEIUuCILgEUShC4IgeARR6IIgCB5BFLogCIJHEIUuCILgEWzrKUpE1QC+SPLjHaBUdPQTMmbv47fxAjLmZMhh5ojFsGxT6KlARGXRMqW8iozZ+/htvICM2WjE5CIIguARRKELgiB4BLcq9Ll2C2ADMmbv47fxAjJmQ3GlDV0QBEFojltn6IIgCEIYotAFQRA8gqMVOhFdSUSfElElEU2O8P5pRLQw+P5HRJRrvZTGomPME4loKxGVE9FqIsqxQ06jiDdezXY3EhETketD3PSMmYhuDn7PFUT0itUyGo2O6zqbiN4joo+D1/YoO+Q0CiJ6gYgOEtEnUd4nInoqeD7KiegHhhyYmR25QOmOtBNAdwAZADYD6BW2zb0A/hr8/xYAC+2W24Ix/whAZvD/X7t5zHrGG9yuLYC1ANYDGGS33BZ8x3kAPgZwZnD9bLvltmDMcwH8Ovh/LwC77ZY7xTFfBuAHAD6J8v4oAO8AIAAXA/jIiOM6eYZ+EYBKZt7FzPUAXgUwNmybsQD+X/D/1wH8mMjV7VjijpmZ32PmuuDqeihNu92Knu8YAKYD+COA41YKZxJ6xvxLAM8w8xEAYOaDFstoNHrGzADOCP7fDs0b0bsKZl4LpR1nNMYCeIkV1gPIIqJOqR7XyQq9M4A9mvW9wdcibsPMjQBqAbS3RDpz0DNmLeOg3OXdStzxEtEAAF2ZeamVgpmInu+4J4CeRPQhEa0noistk84c9Iy5GMDtRLQXSrvL8daIZhuJ/tZ1oaunqE1EmmmHx1jq2cZN6B4PEd0OYBCAYaZKZC4xx0tEaQBmAfi5VQJZgJ7vuAUUs8t/Q3kCKyGiC5m5xmTZzELPmG8F8CIzP0FElwB4OTjmgPni2YIpusvJM/S9ALpq1rug+WPYyW2IqAWUR7VYjzlOR8+YQUSXAygEMIaZT1gkmxnEG29bABcCeJ+IdkOxNS52uWNU73X9T2ZuYObPAXwKRcG7FT1jHgfgNQBg5nUAWkEpYuVVdP3WE8XJCn0DgDwi6kZEGVCcnovDtlkM4GfB/28E8C8OehxcStwxB00Qz0FR5m63rcYcLzPXMnMHZs5l5lwoPoMxzFxmj7iGoOe6fguK8xtE1AGKCWaXpVIai54xVwH4MQAQ0QVQFHq1pVJay2IAdwajXS4GUMvM+1Peq93e4Die4lEAdkDxkBcGX3sEyo8aUL70fwCoBFAKoLvdMlsw5lUADgDYFFwW2y2zmeMN2/Z9uDzKRed3TABmAtgKYAuAW+yW2YIx9wLwIZQImE0ARtotc4rj/TuA/QAaoMzGxwH4FYBfab7jZ4LnY4tR17Wk/guCIHgEJ5tcBEEQhAQQhS4IguARRKELgiB4BFHogiAIHkEUuiAIgkcQhS4IguARRKELgiB4hP8P9K4fbVYm/BEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "util.plot_points(Xb[:,1:3],Yb==1)\n",
    "x1 = np.arange(0, 1, 0.01)\n",
    "x2= np.arange(1,0, -0.01)\n",
    "plt.plot(x1,x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we look at the code, we see that during training the model is using gradient descent with the following gradient:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_grad(X, Y, theta):\n",
    "    \"\"\"Compute the gradient of the loss with respect to theta.\"\"\"\n",
    "    m, n = X.shape\n",
    "\n",
    "    margins = Y * X.dot(theta)\n",
    "    probs = 1. / (1 + np.exp(margins))\n",
    "    grad = -(1./m) * (X.T.dot(probs * Y))\n",
    "\n",
    "    return grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In formulas this means the algorithm is trying to find a zero of the gradient \n",
    "$$ \\nabla_\\theta J(\\theta) = -\\frac 1m \\sum_{i=1}^m \\frac{y^{(i)}x^{(i)}}{1+\\exp(y^{(i)}\\theta^Tx^{(i)})},$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i.e. it is trying to minimize \n",
    "$$ J(\\theta) = -\\frac 1m \\sum_{i=1}^m \\log\\left( \\frac 1{1+\\exp(-y^{(i)}\\theta^Tx^{(i)})}\\right).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But because the data in B is linearly separable, at some point gradient descent will find a $\\theta$ with\n",
    "$$ y^{(i)}\\theta^T x^{(i)} > 0$$\n",
    "for all training examples $(x^{(i)},y^{(i)})$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Such a $\\theta$ can always be improved upon by multiplying $\\theta$ with a large positive constant. Therefore the gradient descent will never converge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (c)\n",
    "\n",
    "*i. Using a different constant learning rate* \n",
    "\n",
    "This will not help, because the argument at the end of (b) would still apply."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*ii. Decreasing the learning rate over time*\n",
    "\n",
    "If the learning rate is decreasing fast enough, then the stop criteria for the implemented gradient descent algorithm would indicate convergence, but mathematically it would not converge to a minimum of the cost function (because such a minimum doesn't exist)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*iii. Linear scaling of the input features*\n",
    "\n",
    "After linear scaling the data would still be linearly separable, so this wouldn't fix the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*iv. Adding a regularization term $\\|\\theta\\|_2^2$ to the loss function*\n",
    "\n",
    "This would lead to convergence, because it punishes the algorithm for lowering the cost function for scaling the components of $\\theta$ indefinitely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*v. Adding zero-mean Gaussian noise to the training data or labels.*\n",
    "\n",
    "As long as this causes the data to become linearly inseparable, it would work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (d)\n",
    "\n",
    "A support vector machine with the hinge loss (and no regularization) would not have problems with datasets like B, because once the model learned how to separate the data perfectly, the hinge loss will be $0$ and the algorithm will stop.\n",
    "\n",
    "If the SVM also has $L_2$-regularization, then it will still work on datasets like B, because then the SVM training would approximately try to minimize the geometric margin, which is not affected by scaling $\\theta$."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
